A digitized image signal generated by a two-dimensional array of sensors consists of pixel values representing the color intensities at each point of the color image. Conventional color photography utilizes three overlapping color sensing layers having sensitivities in different regions of the color spectrum. Typically, red, green and blue are the three selected colors. However, digital cameras are limited to a single array of sensors, so that there is only one “layer.” Within this layer, each sensor determines the intensity of a particular color. As a result, the color array does not produce a color image in the traditional sense, but rather a collection of individual color samples. The assignment of the colors to the individual pixels within the array is sometimes referred to as the color filter array (CFA), or the color “mosaic pattern.” To produce a true color image in which each pixel has a full set of color samples, a substantial amount of interpolation is required in order to estimate the missing information. The interpolation operation is typically referred to as “demosaicing.”
The general concept of demosaicing will be described with reference to FIG. 1. An image sensor array 10 for capturing a digitized image signal is represented by a mosaic pattern that follows an arrangement referred to as a Bayer pattern (i.e., a repeating 2×2 sensor array kernel having two green pixels diagonally positioned within the kernel and having one red sensor and one blue sensor). The sensor array is a color-sampling device that acquires its samples using a number of discrete color-specific detectors. By example, the array may have 1500 columns and 1500 rows, but the numbers of columns and rows are not significant to the overall understanding of the process.
In the demosaicing of the image that is captured by the sensor array 10, the image may be considered as being decomposed into four input image planes 12, 14, 16 and 18. Each input image plane satisfies two conditions. First, the planes have the identical number of samples within a horizontal and vertical sampling interval. Second, all of the samples in the given input image plane have identical color properties, although multiple image planes can have the same color properties. For example, the image planes 12 and 18 have green samples, as indicated by the letter “G.” Image plane 14 includes all of the samples that are specific to the color blue, as indicated by the letter “B,” while image plane 16 includes all of the red samples from the sensor array 10, as indicated by the letter “R.” Alternatively, only three input image planes may be considered, if the two image planes having green samples are combined. While the illustrated embodiment shows a Bayer pattern, other patterns are used by persons skilled in the art.
In FIG. 1, there are three different colors and there are four possible color arrangements within a two-by-two neighborhood of pixels. The number of possible color arrangements is consistent with the number of image planes 12, 14, 16 and 18. It is possible to use twelve convolution kernels (3×4 =12) in the demosaicing process. The twelve convolution kernels are selected on the basis of which of the input image planes contains the sample for the pixel that is to be the current center of a movable neighborhood in which interpolation occurs and on the basis of which monochromatic output plane 20, 22 and 24 is presently being constructed. Each monochromatic output plane contains original color samples and contains interpolated intensity values that are derived from the intensity values of the input image planes.
To generate the missing information at a particular pixel, information from neighboring pixels is used, since there is a statistical dependency among the pixels in the same region of a particular image. Various demosaicing algorithms are available. “Bilinear interpolation” is an approach that attempts to minimize the complexity of the demosaicing operation. A bilinear interpolation algorithm interpolates color intensities only from the same-color sensors. That is, the information from the red sensors is treated independently from the information from the green and blue sensors. Similarly, the information from the green sensors is treated independently from the red and blue sensors, and the information from the blue sensors is treated independently from the outputs of the red and green sensors. To provide a red intensity value at a given pixel, the values measured by the red sensors in a designated size neighborhood (e.g., in a neighborhood of nine sensors having the given pixel as its center) are interpolated. If the mosaic pattern of sensors is a Bayer pattern, the bilinear interpolation algorithm may use twelve kernels of convolution to reconstruct the image, as previously noted.
The bilinear interpolation approach of isolating the different colors in the demosaicing operation requires a relatively low level of computational overhead. The tradeoff is that the reconstructed image often contains significant artifacts. Other interpolation approaches that consider different-color sample values may introduce fewer artifacts, but the cost may be a significant increase in the computational overhead.